Statisticians set sights on observational studies.

tically signifi cant result. In a separate analysis, his team found that the main fi ndings of only one of six highly cited observational studies could be replicated; the conclusions from the other fi ve were refuted. “The combination of selec tive reporting and multiplicity of analysis undermines the credibility of epidemiological studies,” Ioannidis said at the an nual meeting of the American Association for the Advancement of Science in San Francisco. Selective reporting, also called publication bias, comes about partly because so many epidemiological studies are designed to generate new ideas (exploratory trials), not test specifi c hypotheses. The authors analyze the data and then decide what to publish, focusing on the statistically signifi cant results—without mentioning other factors that were tested but that failed to show a signifi cant association. This type of data mining can uncover potentially interesting associations, but the authors need to fully report how they discovered the relationship and explicitly state that the results need to be reproduced in a study designed to examine that hypothesis, Pocock said. “The epidemiological literature by itself is a literature that practically has ubiquitous statistically signifi cant fi ndings,” Ioannidis said. In a survey of 389 papers reporting the results of observational studies, his re search group found that 88% included at least one statistically signifi cant positive association in the abstract, while only 43% reported a nonsignifi cant association. This fi nding is despite the fact that nearly all associations are not expected to be statistically signifi cant. That means there are a lot of nonsignifi cant associations that are not being reported, despite having been tested. “In the meta-analysis literature, where one pools fi ndings across studies, the Statisticians Set Sights on Observational Studies